Rapid Interactive Installation Development Using Robust Computer Vision and Image-Based Rendering

The paper describes a technique which lets designers implement artistic ideas rapidly into an autonomous interactive art system. The technique consists of two parts: the fixed set of computer vision algorithms and image-based rendering with branching animation sequences. Despite some limitations, the technique provides easy implementation of the wide range of fascinating interactive scenes. The availability of low-cost desktops, compact computing systems, digital cameras and TV panels together with existing sophisticated algorithms and software provides an opportunity for spreading the autonomous interactive art installations widely into everyday life. A lot of systems and toolkits are available for developing and deploying such installations [1]. But it is hard to find a system which provides the designer with a fast and efficient way to implement artistic ideas into the interactive form. So there is a demand for a software platform which would allow to implement a wide range of the interactive scenes in a simple way. In this paper, the technique for constructing such a platform is proposed. Usually an autonomous interactive installation plays some scene while perceiving and responding to a user’s activity. We will consider a video interactive installation with hardware consisting of video input device (video camera), a processor (desktop or compact computing device) and a video output device (TV panel). The key modules for interactive installation software are: 1. Video Analysis: interpretation of user’s actions. It is possible to construct a robust computer vision module, working in uncontrolled light conditions and performing following tasks [2]: – detect objects appearing and disappearing; – detect objects’ motion direction and speed; – recognize human face and its expression; – recognize several hand gestures. The output data of the module is enough for many camera-based interactive scenarios. 2. Behaviour & Dynamics: scene internal state representation and changing accordingly the user’s actions. 3. Render: visual representation of the scene state. We propose to use imagebased rendering [3] as the universal and simple way for utilizing the animation obtained with different methods: 2d, 3d animation and live video recordings. More specifically, it is proposed to represent the object’s dynamics by a number of animation sequences. This sequences have common frames, in which it is possible to switch between the sequences while playing. Such sequences can be called “branching animation sequences” (Fig. 1). Fig. 1. Four Branching Animation Sequences. The experimental platform for testing the proposed technique was developed. A number of interactive installations were implemented. The experiments proved the simplicity of implementing the interactive scenes. It seems that the capabilities are quite enough for the mass production of the wide diversity of the simple interactive installations. Part of the research was supported by the grant 09-01-00523 from the Russian Foundation for Basic Research and the Presidium of the Russian Academy of Sciences fundamental program 29, project P(29)7-2. The author would like to thank G. Malyshev and P. Zakrevskiy for 3d animation, S. Zamuraev for fruitful discussions, I. Ilyin for the help in programming, S. Perevalova for the help in demo, A. Poptsova for the text correction.